Middleware benchmarking: approaches, results, experiences

The report summarizes the results of the Workshop on Middleware Benchmarking held during OOPSLA 2003. The goal of the workshop was to help advance the current practice of gathering performance characteristics of middleware implementations through benchmarking. The participants of the workshop have focused on identifying requirements and obstacles of middleware benchmarking and forming a position on the related issues. Selected requirements and obstacles are presented in section 2 of this report, together with guidelines to adhere to when benchmarking in section 3, open issues of current practice in section 4, and perspectives on further research in section 5.